System and method for capture and rendering of performance on synthetic string instrument

ABSTRACT

Synthetic multi-string musical instruments have been developed for capturing and rendering musical performances on handheld or other portable devices in which a multi-touch sensitive display provides one of the input vectors for an expressive performance by a user or musician. Visual cues may be provided on the multi-touch sensitive display to guide the user in a performance based on a musical score. Alternatively, or in addition, uncued freestyle modes of operation may be provided. In either case, it is not the musical score that drives digital synthesis and audible rendering of the synthetic multi-string musical instrument. Rather, it is the stream of user gestures captured at least in part using the multi-touch sensitive display that drives the digital synthesis and audible rendering.

CROSS-REFERENCE TO RELATED APPLICATION(S)

The present application claims the benefit of U.S. ProvisionalApplication No. 61/411,900, filed Nov. 9, 2010, the entirety of which isincorporated herein by reference.

BACKGROUND

1. Field of the Invention

The invention relates generally to musical instruments and, inparticular, to techniques suitable for use in portable device hostedimplementations of musical instruments for capture and rendering ofmusical performances.

2. Description of the Related Art

The field of mobile music has been explored in several developing bodiesof research. See generally, G. Wang, Designing Smule's iPhone Ocarina,presented at the 2009 on New Interfaces for Musical Expression,Pittsburgh (June 2009) and published athttps://ccrma.stanford.edu/-ge/publish/ocarina-nime2009.pdf. Oneapplication of this research has been the Mobile Phone Orchestra(MoPhO), which was established in 2007 at Stanford University's Centerfor Computer Research in Music and Acoustics and which performed itsdebut concert in January 2008. The MoPhO employs more than a dozenplayers and mobile phones which serve as a compositional and performanceplatform for an expanding and dedicated repertoire. Although certainlynot the first use of mobile phones for artistic expression, the MoPhOhas been an interesting technological and artistic testbed forelectronic music composition and performance. See generally, G. Wang, G.Essl and H. Penttinen, MoPhO: Do Mobile Phones Dream of ElectricOrchestras? in Proceedings of the International Computer MusicConference, Belfast (August 2008).

Mobile phones and other portable computing devices are growing in sheernumber and computational power. Hyper-ubiquitous and deeply entrenchedin the lifestyles of people around the world, they transcend nearlyevery cultural and economic barrier. Computationally, the mobile phonesand pad-type devices of today offer speed and storage capabilitiescomparable to desktop computers from less than ten years ago, renderingthem surprisingly suitable for real-time sound synthesis and othermusical applications. Like traditional acoustic instruments, the mobilephones are intimate sound producing devices. By comparison to mostinstruments, they are somewhat limited in acoustic bandwidth and power.However, mobile phones have the advantages of ubiquity, strength innumbers, and ultramobility, making it feasible to hold jam sessions,rehearsals, and even performance almost anywhere, anytime.

Research to practically exploit such devices has been ongoing for sometime. For example, a touch-screen based interaction paradigm withintegrated musical synthesis on a Linux-enabled portable device such asan iPaq™ personal digital assistant (PDA) was described by Geiger. SeeG. Geiger, PDa: Real Time Signal Processing and Sound Generation onHandheld Devices, in Proceedings of the International Computer MusicConference, Singapore (2003); G. Geiger, Using the Touch Screen as aController for Portable Computer Music Instruments in Proceedings of theInternational Conference on New Interfaces for Musical Expression, Paris(2006). Likewise, an accelerometer based custom-made augmented PDAcapable of controlling streaming audio was described by Tanaka. See A.Tanaka, Mobile Music Making, in Proceedings of the 2004 Conference onNew Interfaces for Musical Expression, pages 154-156 (2004).

Indeed, use of mobile phones for sound synthesis and live performancewas pioneered by Schiemer in his Pocket Gamelan instrument, seegenerally, G. Schiemer and M. Havryliv, Pocket Gamelan: TuneableTrajectories for Flying Sources in Mandala 3 and Mandala 4, inProceedings of the 2006 Conference on New Interfaces for MusicalExpression, pages 37-42, Paris, France (2006), and remains a topic ofresearch. The MobileSTK port of Cook and Scavone's Synthesis Toolkit(STK) to Symbian OS, see G. Essl and M. Rohs, Mobile STK for Symbian OS,in Proceedings of the International Computer Music Conference, NewOrleans (2006), was perhaps the first full parametric synthesisenvironment suitable for use on mobile phones. Mobile STK was used incombination with accelerometer and magnetometer data in ShaMus to allowpurely on-the-phone performance without any laptop. See G. Essl and M.Rohs, ShaMus—A Sensor-Based Integrated Mobile Phone Instrument, inProceedings of the International Computer Music Conference, Copenhagen(2007).

As researchers seek to transition their innovations to commercialapplications deployable to modern handheld devices such as iPad™ andiPhone® mobile digital devices(available from Apple Inc.) and otherplatforms operable within the real-world constraints imposed byprocessor, memory and other limited computational resources thereofand/or within communications bandwidth and transmission latencyconstraints typical of wireless networks, practical challenges present.

Improved techniques and solutions are desired.

SUMMARY

It has been discovered that, despite practical limitations imposed bymobile device platforms and applications, truly captivating musicalinstruments may be synthesized in ways that allow musically expressiveperformances to be captured and rendered in real-time. In some cases,the synthetic musical instruments can transform the otherwise mundanemobile devices into social instruments that facilitate performances inco-located ensembles of human performers and/or at distances that fostera unique sense of global connectivity.

Accordingly, techniques have been developed for capturing and renderingmusical performances on handheld or other portable devices using signalprocessing techniques suitable given the somewhat limited capabilitiesof such devices and in ways that facilitate efficient encoding andcommunication of such captured performances via wireless networks. Thedeveloped techniques facilitate the capture, encoding and use of gesturestreams for rendering of a musical performance. In some embodiments, agesture stream encoding facilitates audible rendering of the musicalperformance locally on the portable device on which the musicalperformance is captured, typically in real time. In some embodiments, agesture stream efficiently codes the musical performance fortransmission from the portable device on which the musical performanceis captured to (or toward) a remote device on which the musicalperformance is (or can be) rendered. Indeed, is some embodiments, agesture stream so captured and encoded may be rendered both locally andon remote devices using substantially identical or equivalent instancesof a digital synthesis of the musical instrument executing on the localand remote devices.

In general, rendering includes synthesis of tones, overtones, harmonics,perturbations and amplitudes and other performance characteristics basedon the captured (and often transmitted) gesture stream. In some cases,rendering of the performance includes audible rendering by converting toacoustic energy a signal synthesized from the gesture stream encoding(e.g., by driving a speaker). In some cases, the audible rendering is onthe very device on which the musical performance is captured. In somecases, the gesture stream encoding is conveyed to a remote devicewhereupon audible rendering converts a synthesized signal to acousticenergy.

Thus, in some embodiments, a synthetic musical instrument (such as asynthetic violin, guitar or other multi-string instrument) allows thehuman user to control an expressive physical model or wavetablesynthesis of a vibrating string and resonant body, using a multi-touchsensitive display to express string length and/or excitation indicativegestures (e.g., bowing, strumming, fingers on strings, plucking ofstrings, damping, etc.). The user actually causes the sound and controlsthe parameters affecting pitch, timbre, quality, etc. When a user playswith a musical score, which may also provide accompaniment, the visualcues generated based on the musical score suggest to the player what andhow to play and when (typically with fingering positions and noteextents, but in some embodiments, also by marking gestures andperformance figures such as vibrato, trills, etc.). The user is free togo off and improvise, double the intended notes at the octave or otherinterval, do other ornaments, play spontaneous counterpoint, etc. Theycan also opt to play the notes and figures indicated as accurately aspossible, to achieve a high score. When combined with the optionalright-hand iPhone (device B) bowing controller, the user has even moreexpressive power, and more responsibility for the performance.

In some embodiments, a storybook mode provides lesson plans which teachthe user to play the synthetic instrument and exercise. Userperformances may be graded (or scored) as part of a game (orsocial-competitive application framework), and/or as a proficiencymeasure for advancement from one stage of a lesson plan to the next. Ingeneral, better performance lets the player (or pupil) advance faster.High scores both encourage the pupil (user) and allow the system to knowhow quickly to advance the user to the next level and, in some cases,along which game or instructive pathway. In each case, the user isplaying a real/virtual physical model of an instrument, and theirgestures actually control the sound, timing, etc.

Often, both the device on which a performance is captured and that onwhich the corresponding gesture stream encoding is rendered areportable, even handheld devices, such as pads, mobile phones, personaldigital assistants, smart phones, media players, book readers, laptop ornotebook computers or netbooks. In some cases, rendering is to aconventional audio encoding such as AAC, MP3, etc. In some cases,rendering to an audio encoding format is performed on a computationalsystem with substantial processing and storage facilities, such as aserver on which appropriate CODECs may operate and from which contentmay thereafter be served. Often, the same gesture stream encoding of aperformance may (i) support local audible rendering on the capturedevice, (ii) be transmitted for audible rendering on one or more remotedevices that execute a digital synthesis of the musical instrumentand/or (iii) be rendering to an audio encoding format to supportconventional streaming or download.

In some embodiments in accordance with the present invention(s), amethod includes using a first portable computing device as a syntheticstring instrument; presenting on a multi-touch sensitive display of theportable computing device, and in correspondence with a musical score,temporally synchronized visual cues relative to respective strings ofthe synthetic string instrument; capturing user gestures indicative oflength of respective strings of the synthetic string instrument fromdata sampled in correspondence with respective finger contacts with themulti-touch sensitive display along visual depictions of the respectivestrings; and capturing user gestures indicative of excitation of atleast one of the strings. The method further includes encoding a gesturestream for a performance of the user by parameterizing at least a subsetof the string length and string excitation indicative user gestures; andaudibly rendering the performance on the portable computing device usingthe encoded gesture stream as an input to a digital synthesis of thesynthetic string instrument executing on the first portable computingdevice, wherein the captured gesture stream, and not the musical scoreitself, drives the digital synthesis. In some embodiments, the stringexcitation indicative gestures are distinct from the string lengthindicative gestures.

In some embodiments, the digital synthesis includes a model of acousticresponse for one of a violin, a viola, a cello and a double bass; andthe captured string excitation indicative gestures include a bowtraversal gesture indicated by contact of the user's thumb or fingerwith a rotating visual on the multi-touch sensitive display. In somecase, radial excursion of the user's thumb or finger contact with therotating visual is indicative of bow speed. In some cases, the capturedstring excitation indicative gestures further include pluck- orstrum-type gestures indicated by contact of the user's thumb or fingerwith respective visual indications of the strings.

In some embodiments, the digital synthesis is of one of a guitar, banjo,ukulele, lute or setar; and the captured string excitation indicativegestures include pluck- or strum-type gestures indicated by contact ofthe user's thumb or finger with respective visual indications of thestrings. In some cases, the digital synthesis includes wavetable orfrequency modulation synthesis in correspondence with theparameterization of at least the string length indicative user gestures.In some cases, the digital synthesis includes physical model of acousticresponse for the guitar, banjo, ukulele, lute or setar.

In some embodiments, the method further includes, responsive to a userselection on the multi-touch sensitive display, retrieving an encodingof the musical score via the communications interface.

In some embodiments, the method further includes grading the user'sperformance based at least in part on correspondence of captured fingercontact gestures with notes or positions and timings encoded in themusical score. In some embodiments, the method further includes gradingthe user's performance based at least in part on correspondence ofcaptured finger contact gestures with vibrato, trilling or otherspatio-temporal effects encoded in the musical score.

In some embodiments, the method further includes presenting on themulti-touch sensitive display a lesson plan of exercises, wherein thecaptured gesture stream corresponds to performance by the user of aparticular one of the exercises; and advancing the user to a nextexercise of the lesson plan based on a grading of the user's performanceof the particular exercise.

In some embodiments in accordance with the present invention(s), amethod includes using a first portable computing device as a syntheticstring instrument, capturing user gestures relative to respectivestrings of the synthetic string instrument from data sampled incorrespondence with respective finger contacts with a multi-touchsensitive display of the portable computing device and capturing usergestures indicative of bow traversal of at least one of the strings. Agesture stream is encoded for a performance of the user byparameterizing at least a subset of events captured from the fingercontacts and bow traversal, the performance is then audibly rendered onthe portable computing device using the encoded gesture stream as aninput to a digital synthesis of the synthetic string instrumentexecuting on the first portable computing device. In some cases,temporally synchronized visual cues are presented on the multi-touchsensitive display in correspondence with a musical score to guide theuser's gestures relative to the respective strings of the syntheticstring instrument. The gesture stream captured from the user'sperformance, and not the musical score itself, drives the digitalsynthesis.

In some cases, user gestures indicative of bow traversal are captured incorrespondence with thumb or finger contact with the multi-touchsensitive display. In some cases, the method includes dynamicallycorrecting captured finger contact gestures in accord with notes orpositions encoded in the musical score.

In some cases, the method includes grading the user's performance basedat least in part on correspondence of captured finger contact gestureswith notes or positions and timings encoded in the musical score. Insome cases, the method includes grading the user's performance based atleast in part on correspondence of captured finger contact gestures withvibrato, trilling or other spatio-temporal effects encoded in themusical score.

In some embodiments, the method further includes manipulating a secondportable device relative to the first, wherein the user gesturesindicative of bow traversal include lateral movement and inclination ofthe second portable device captured in correspondence with themanipulation. In some cases, the second portable computing deviceincludes either or both of multi-axis accelerometer and a gyroscopicsensor for capture of orientation and motion dynamics of the secondportable computing device. In such cases, the second portable computingdevice is configured to compute and wirelessly communicate to the firstportable computing device the bow traversal gestures based on thecaptured orientation and motion dynamics. In some cases, the methodincludes capturing at a multi-touch sensitive display of the secondportable computing device further user gestures indicative of one ormore of bow pressure and bow-on-string.

In some cases, the method further includes grading the user'sperformance based at least in part on correspondence of one or moreparameterizations of the captured bow traversal gestures with bowingtechniques encoded in the musical score, wherein the one or moreparameterizations are selected from the set of speed of travel, positionof a sounding point, and method of attack.

In some cases, the method includes presenting on the multi-touchsensitive display a lesson plan of exercises, wherein the capturedgesture stream corresponds to performance by the user of a particularone of the exercises and advancing the user to a next exercise of thelesson plan based on a grading of the user's performance of theparticular exercise.

In some cases, the first portable computing device includes acommunications interface, and the method further includes transmittingthe encoded gesture stream via the communications interface forrendering of the performance on a remote device.

In some cases, the digital synthesis includes a model of acousticresponse for one of a violin, a viola, a cello and a double bass. Insome cases, the first and/or second portable computing devices are eachselected from the group of: a compute pad, a laptop computer, notebookcomputer or netbook; a personal digital assistant or book reader; and amobile phone or media player. In some cases, the remote device includesa server from which the rendered performance is subsequently supplied asone or more audio encodings thereof.

In some cases, the method further includes audibly rendering a secondperformance on the portable computing device using a second gesturestream encoding received via the communications interface directly orindirectly from a second remote device, the second performance renderingusing the received second gesture stream encoding as an input to thedigital synthesis of the musical instrument. In some cases, the methodfurther includes geocoding the transmitted gesture stream and displayinga geographic origin for, and in correspondence with audible renderingof, a third performance encoded as a third gesture stream received viathe communications interface directly or indirectly from a third remotedevice.

In some embodiments, a computer program product is encoded in one ormore non-transitory media, the computer program product includinginstructions executable on a processor of the portable computing deviceto cause the portable computing device to perform one or more of theaforementioned methods. In some cases, media are readable by theportable computing device or readable incident to a computer programproduct conveying transmission to the portable computing device.

In some embodiments, an apparatus includes a portable computing devicehaving a multi-touch display interface and machine readable codeexecutable on the portable computing device to implement a syntheticmusical instrument. The machine readable code includes instructionsexecutable to capture both: (i) user gestures relative to respectivestrings of the synthetic string instrument from data sampled incorrespondence with respective finger contacts with a multi-touchsensitive display of the portable computing device and (ii) usergestures indicative of bow traversal of at least one of the strings andto encoded a gesture stream for a performance of the user byparameterizing at least a subset of events captured from the fingercontacts and bow traversal. The machine readable code is furtherexecutable to audibly render the performance on the portable computingdevice using the encoded gesture stream as an input to a digitalsynthesis of the synthetic string instrument executing on the portablecomputing device.

In some cases, the apparatus is configured to wirelessly communicatewith a second portable computing device proximate thereto. The secondportable computing device includes either or both of multi-axisaccelerometer and a gyroscopic sensor for capture of orientation andmotion dynamics of the second portable computing device. Machinereadable code is executable on the second portable computing device tocompute and wirelessly communicate to the first portable computingdevice the bow traversal gestures based on the captured orientation andmotion dynamics. In some cases, the apparatus is embodied as one or moreof a compute pad, a handheld mobile device, a mobile phone, a laptop ornotebook computer, a personal digital assistant, a smart phone, a mediaplayer, a netbook, and a book reader.

Finally, in some embodiments a computer program product is encoded innon-transitory media and includes instructions executable to implement asynthetic musical instrument on a portable computing device having amulti-touch display interface. In particular, the computer programproduct encodes instructions executable to capture both: (i) usergestures relative to respective strings of the synthetic stringinstrument from data sampled in correspondence with respective fingercontacts with a multi-touch sensitive display of the portable computingdevice and (ii) user gestures indicative of bow traversal of at leastone of the strings and to encode a gesture stream for a performance ofthe user by parameterizing at least a subset of events captured from thefinger contacts and bow traversal The computer program product encodesfurther instructions executable to audibly render the performance on theportable computing device using the encoded gesture stream as an inputto a digital synthesis of the synthetic string instrument executing onthe portable computing device.

These and other embodiments in accordance with the present invention(s)will be understood with reference to the description herein as well asthe drawings and appended claims which follow.

BRIEF DESCRIPTION OF THE DRAWINGS

The present invention is illustrated by way of example and notlimitation with reference to the accompanying figures, in which likereferences generally indicate similar elements or features.

FIGS. 1 and 2 depict performance uses of a portable computing devicehosted implementation of a synthetic violin in accordance with someembodiments of the present invention. FIG. 1 depicts an individualperformance use and FIG. 2 depicts performances as an ensemble.

FIG. 3 illustrates certain aspects (including visual cuing and notesounding gestures) of a user interface design for a synthetic violininstrument in accordance with some embodiments of the present invention.

FIGS. 4A, 4B and 4C illustrate spatio-temporal cuing aspects of a userinterface design for a synthetic violin instrument in accordance withsome embodiments of the present invention. FIG. 4A illustrates a pair oftemporally sequenced note cues presented in correspondence with anunderlying score. FIG. 4B illustrates a pair of temporally sequencednote cues, together with a vibrato cue for the current note. FIG. 4Cillustrates a current note cue, together with a vibrato cue for thecurrent note and a desired user finger contact position inspatio-temporal correspondence with the underlying score. Each of thefigures illustrate a touch screen bowing artifact or visual device,wherein in accord with some embodiments of the present invention,finger/thumb contact gestures are indicative of bow contact and radialposition codes bow traversal speed.

FIG. 5 is a functional block diagram that illustrates capture andencoding of user gestures corresponding to several notes of aperformance on a synthetic violin instrument and acoustic rendering ofthe performance in accordance with some embodiments of the presentinvention.

FIG. 6 is a functional block diagram that illustrates capture, encodingand transmission of a gesture stream encoding corresponding to a userperformance on a synthetic violin instrument together with receipt ofthe gesture stream encoding and acoustic rendering of the performance ona remote device.

FIG. 7 is a functional block diagram that illustrates capture andencoding of user gestures corresponding to several chords of aperformance on a synthetic guitar instrument and acoustic rendering ofthe performance in accordance with some embodiments of the presentinvention.

FIG. 8 illustrates certain aspects of a user interface design for astorybook teaching mode of a synthetic violin instrument in accordancewith some embodiments of the present invention.

FIG. 9 illustrates features of a mobile device that may serve as aplatform for execution of software implementations in accordance withsome embodiments of the present invention.

FIG. 10 is a network diagram that illustrates cooperation of exemplarydevices in accordance with some embodiments of the present invention.

Skilled artisans will appreciate that elements or features in thefigures are illustrated for simplicity and clarity and have notnecessarily been drawn to scale. For example, the dimensions orprominence of some of the illustrated elements or features may beexaggerated relative to other elements or features in an effort to helpto improve understanding of embodiments of the present invention.

DESCRIPTION

Synthetic multi-string musical instruments have been developed forcapturing and rendering musical performances on handheld or otherportable devices in which a multi-touch sensitive display provides oneof the input vectors for an expressive performance by a user/musician(hereafter the user). Visual cues may be provided on the multi-touchsensitive display to guide the user in a performance based on a musicalscore. Alternatively, or in addition, uncued freestyle modes ofoperation may be provided. In either case, it is not the musical scorethat drives digital synthesis and audible rendering of the syntheticmulti-string musical instrument. Rather, it is the stream of usergestures captured at least in part using the multi-touch sensitivedisplay that drives the digital synthesis and audible rendering. In somecases or embodiments, additional sources of user gestures, such asdevice motion captured by way of an accelerometer embedded in theportable device or proximate motion of another communicating device maycontribute to the gesture stream that drives the digital synthesis andaudible rendering.

In these ways and for some embodiments (particularly implementations ofa bowed string instrument such as a violin, viola, cello, double bass,etc.), the user is able to control an actual expressive physical modelof a string and resonant body using gestures captured on a multi-touchsensitive display (e.g., bowing, fingers on strings, plucking ofstrings, damping, etc.). For some embodiments (particularly those inwhich excitation of string vibration is by way of an impulse (e.g., aguitar, banjo, ukulele, lute, setar, etc.), wavetable synthesistechniques may be employed as a computationally efficient and attractivesynthesis technique.

In either case, the user is actually causing the sound and controllingthe parameters affecting pitch, timbre, quality, etc. When playing witha musical score, which may also provide accompaniment, the scorecontains indications to the user of what and how to play and when(typically with fingering positions and note extents, but in someembodiments, also marking gestures and performance figures such asvibrato, trills, string bending, etc.). The user is free to go off scoreand improvise double the intended notes at the octave or other interval,do other ornaments, play spontaneous counterpoint, etc. In game orlesson plan modes, the user may opt to play the notes and performancefigures indicated as accurately as possible, so as to achieve a highscore or to demonstrate ability to move on in a lesson plan.

In some cases, musical scores and/or lesson plans may be retrieved by auser via a communications interface as an “in-app” purchase. Generally,scores may be encoded in accord with amy suitable coding scheme such asin accord with well known musical instrument digital interface (MIDI)standards, file formats and protocols (e.g., standard MIDI [.mid or.smf] formats, MIDI karaoke [.kar] formats; extensible music file, XMFformats; extensible MIDI [.xmi] formats; RIFF-based MIDI [.rmi] formats;extended RMID formats, etc.). In some cases, user performances may beuploaded and shared via such a communications interface. Often,performance uploads are encoded as the very gesture streams used todrive the digital synthesis and audible rendering on the local portabledevice. In some cases or additionally, local audible rendering is to aconventional audio encoding such as AAC, MP3, etc. In some cases,rendering to an audio encoding format is performed on a remotecomputational system with substantial processing and storage facilities,such as a server on which appropriate CODECs may operate and from whichcontent may thereafter be served.

Much of the description herein emphasizes musical performanceterminology, physical instrument analogs, user interface constructs andrelevant gesture sets characteristic of a synthetic violinimplementation wherein modeled strings are excited with a bow.Nonetheless, based on the description herein, persons of ordinary skillin the art will appreciate suitable modifications and/or extensions forother synthetic multi-string instruments. In particular, terminology,physical instrument analogs, user interface constructs and relevantgesture sets for multi-string instruments (including the describedviolin/fiddle, as well as guitar-type synthetic instruments) in whichnotes and chords are sounded using pluck- or strum-type excitation andgestures will also be understood. Likewise, although device, operatingsystem and development platform aspects typical of iOS devices such asthe iPad, iPod Touch and iPhone handhelds (available from Apple, Inc.)are emphasized, nothing herein shall be taken as limiting to aparticular device, operating system or platform. iOS, iPad, iPod andiPhone are all trademarks of Apple, Inc. In view of the foregoing, andwithout limitation, we now describe certain illustrative embodiments.

FIGS. 1 and 2 depict performance uses of a portable computing devicehosted implementation of a synthetic violin in accordance with someembodiments of the present invention. In particular, FIG. 1 depicts anindividual performance use of a pad-type computing device programmed toimplement a simplified three (3) string synthetic violin. FIG. 2 depictsperformances of multiple user musicians as an ensemble (on respectiveportable computing device hosted implementations of the syntheticviolin).

User contacts with visual depictions of individual strings are capturedusing a multi-touch sensitive display, and these captured fingercontacts constitute string length indicative gestures that are, in turn,fed to a digital synthesis of acoustic response of the violin togetherwith excitation indicative gestures. Voicing of the synthetic violin isbased on characteristics (materials, resonances, etc.) of the variousacoustically significant components modeled. In this way, the user'sfinger contacts and, indeed, movement of finger contacts along avisually depicted string, dynamically vary the effective length ofparticular string as modeled and, accordingly, the frequency components(including harmonics) of acoustic energy digitally synthesized based onexcitation.

Various excitation gestures may be captured and used to drive thedigital synthesis. For example, in some embodiments, an additionalfinger (or thumb) contact with a rotating display feature provides theuser interface analog of bow contact and transverse travel over one ormore strings of the synthetic musical instrument. Location of suchfinger contact (i.e., closer to, or further from, the center of therotating display feature) codes velocity of bow travel and affectsmodeled excitation of a particular string (or strings). In someembodiments, the breadth of the contact surface, the transient naturecontact or other captured gestures may code bowing techniques thataffect tone quality of the digital synthesis. In some embodiments,finger contacts along a portion of a particular string (as visuallydepicted on the multi-touch sensitive display) are captured aspluck-type excitation gestures. Together, the string length indicativegestures and the excitation indicative gestures are supplied to thedigital synthesis as a gesture stream.

FIG. 3 (together with FIGS. 4A, 4B and 4C) illustrates certainadditional aspects of a user interface design for a synthetic violininstrument in accordance with some embodiments of the present invention.In particular, FIG. 3 depicts user interaction in a songbook guided modein which spatio-temporal cuing of note selection (i.e., of string lengthindicative gestures) and duration of a note sounding is provided inaccordance with a musical score. The user is guided to particular stringlength indicative contact points on the multi-touch sensitive, which(upon actual finger contact) results in captured string lengthindicative gestures that are supplied to the digital synthesis. Forexample, scored coded note cue 301 guides the user to contact the visualdepiction of string 312 at a particular length therealong consistentwith note to be sounded in conjunction with a bowing- or plucking-typeexcitation indicative gesture. A “wave” depiction of vibrato cue 302 onthe contacted string 312 is evocative of vibrato, which the user maygesture with slight back-and-forth movement (or wobble) of his/herfinger contact. As previously described, it is notable that the actualstream of user gestures (here the left hand, ring finger contact, wobbleand release along string 312 and the thumb contact, radial movement andrelease of the right hand on the rotating visual device 321) are whatdrive the digital synthesis as string length indicative and excitationindicative gestures. In a graded or lesson plan mode, notes are “scored”as hit when the contact point, timing and duration corresponds to visualcuing. Actual audible rendering is based on the actual gesture stream(including any premature or late bowing or string contact, any tonalvariance from the musical score based on actual contact point, anyvibrato wobble, or any acceleration/deceleration of the bow based onthumb movement).

FIGS. 4A, 4B and 4C further illustrate (using a sequence of screenimages) spatio-temporal cuing aspects of a user interface design for asynthetic violin instrument in accordance with some embodiments of thepresent invention. FIG. 4A illustrates a pair of temporally sequencednote cues (401, 402) presented in correspondence with an underlyingscore. The first to arrive note cue 401 suggests to the user arelatively short sounding of a note to be expressed by an F# indicativefinger contact on the first string together with a bow contactindicative thumb or finger contact with the rotating bowing device 421.FIG. 4B illustrates a pair of temporally sequenced note cues, togetherwith a vibrato cue 405 for the current note. FIG. 4C illustrates a notecue 403 suggestive of a relatively long expression of an E indicativefinger contact on the second string, together with a vibrato cue 406 forthe current note and a desired user finger contact position (432) inspatio-temporal correspondence with the underlying score.

Each of the aforementioned drawings (FIGS. 4A, 4B and 4C) illustrate atouch screen bowing device 421, wherein in accord with some embodimentsof the present invention, finger/thumb contact gestures are indicativeof bow contact and radial position codes bow traversal speed. Othervisual devices may be employed to facilitate capture of excitationindicative gestures that parameterize bow contact, inclination, pressureand/or speed of travel. The illustrated visual device is but onesuitable example. Likewise, although vibrato cues have been illustratedas visual ornamentation of a corresponding string, other visual cueforms may be employed such as in correspondence with note or chordsounding cues, a visual cuing entirely separate from string depictionsand note/chord cues, as haptic feedback, etc.

FIG. 5 is a functional block diagram that illustrates capture andencoding of user gestures corresponding to several notes of aperformance on a synthetic violin instrument and acoustic rendering(511) of the performance in accordance with some embodiments of thepresent invention. In particular, and as more particularly illustratedabove relative to FIGS. 3, 4A, 4B and 4C, string, a bowing device, andscore-coded visual cues are depicted on display 514 of a portablecomputing device 501 with a multi-touch sensitive display (hereillustrated at form factor reminiscent of an iPad handheld, but moregenerally understood to be consistent with iPhone, iPod Touch or otherportable computing devices at pad, phone, book reader or media playerform factors). Consistent with the role in such devices of a multi-touchsensitive display as both an input and an output device, the multi-touchsensitive display (together with illustrative depictions thereon) isindicated in multiple places and roles thorough the drawing using auniform reference 514.

As will be appreciated by persons of ordinary skill in the art,functionality illustrated as Magic Fiddle Application 550 includesapplication specific code executable on a processor of portablecomputing device 501 as well as firmware, operating system and/orembedded devices thereof. In general, particular allocations offunctionality to application code, to firmware, to operating systemconstructs or to hardware of the portable computing device itself arematters of design choice and accordingly, any illustrated or describedallocation (express or implied) may be subject to further designvariation without departing from the spirit or scope of inventionsdescribed and claimed herein.

In the illustration of FIG. 5, a visual cue generator 557 (operableduring songbook or lesson plan modes of operation) retrieves musicalscore information from storage 556 and, based thereon, supplies asequence of visual cues to be presented on display 514 in correspondencewith string and bowing device visuals. Although a user may (in general)express and sound whatever notes he/she desires by way of string lengthindicative and excitation indicative gestures on multi-touch sensitivedisplay 514, often the user expresses (or at least attempts to express)a sequence of gestures (518) generally in correspondence with the visualcues and the musical score from which such visual cues are generated(557). In some cases, the musical score is demand supplied from a remoteserver or service via wireless data communications 512.

Depending on the implementation (or mode of operation) a variety ofinputs (552) may feed the capture/encoding (553) to produce a gesturestream 551 supplied to synthesis 554. String length indicative gestures(e.g., finger contacts, vibrato indicative finger contact wobble,lateral travel of a finger contact along a string, etc.) and excitationindicative gestures (e.g., finger contacts in a string pluck region,thumb/finger contact at a rotating bowing device andaccelerating/deceleration indicative travel thereacross) captured frommulti-touch sensitive display 514 are collectively summarized asgestures 518. However, additional sources of user gestures may besupported in some embodiments. For example, in some embodiments, anaccelerometer 517 input may be used to capture gestures (519) indicativeof vibrato, trilling or some other ornamentation of the user'sperformance. In some embodiments, manipulation of a second portabledevice (e.g., handheld 509) may be detected and captured usingaccelerometer(s) onboard the second device and communicated to portablecomputing device 501 (and application 550 executing thereon) via a wiredor wireless communications facility such as Bluetooth® communication.Bluetooth is a trademark of the Bluetooth special interest group (SIG).

Building upon gestures captured from multi-touch sensitive display 514and suitably encoded (553), a gesture stream 551 is used to drivedigital synthesis 554. In some embodiments, such a gesture stream 551 isstored (e.g., in storage 556) for possible or optional communication toa remote server, service platform or rendering device as part of asocial media or collaboration interaction. Depending on the nature ofdigital synthesis, data communications and/or remote rendering,differing encodings or subset/superset gesture streams may be employed.In some embodiments, an encoding of the same gesture stream or even thesame encoding of the same gesture stream may drive local digitalsynthesis 554 and audible rendering as well as a remote audiblerendering or rendering to an audio encoding format.

Turning to digital synthesis 554, a variety of computational techniquesmay be employed and will be appreciated by persons of ordinary skill inthe art. For example, exemplary techniques include (i) physical modelingof string vibration and acoustics of the modeled instrument and (ii)wavetable synthesis. In general, for certain types of stringinstruments, particularly the synthetic violin with bowed excitationused herein as a motivating example, physical modeling may provide alivelier, more expressive synthesis that is responsive (in ways similarto physical analogs) to the continuous and expressively variableexcitation of constituent strings afforded by transverse bow travel andto the instantaneous variation of effective string length possible in anon-fretted string instrument. A suitable physical model based musicaltone synthesis system is described in U.S. Pat. No. 5,587,548 namingJulius O. Smith, III as inventor, which is incorporated herein byreference for the limited purpose of describing suitable commutedsynthesis-type (commuted synthesis of string and resonator)implementations of synthesis 554. Additional options include moreliteral physical models, which are generally less controllable but muchmore expressive. Techniques detailed in U.S. Pat. Nos. 5,448,010 and/or5,212,334, each naming Julius O. Smith, III as inventor, or in J. O.Smith, Efficient Simulation of the Reed-Bore and Bow-String Mechanisms,in Proceedings of the 1986 International Computer Music Conference, TheHague, pp. 275-280 (1986) are also suitable for some implementations inaccordance with the present invention(s).

In an embodiment based on a commuted synthesis-type technique describedin the '548 Patent, particular excitation indicative gestures capturedfrom 514 (e.g., bow-on-string indicative finger/thumb contact andtraversal at a given speed or string plucking) are selective for acorresponding table-coded excitation signals e (n). In some embodiments,different tables (i.e., different excitation signals) may used for upbowing and for down bowing. Likewise, single vs. multi-string contactindicative gestures, attack gestures, and bow pressure indicativegestures (if supported) may be selective for respective and differentexcitation signals. In some embodiments, a table-coded excitationsignal, e (n), may be up/down sampled to shift the apparent body size ofthe instrument. In general, filtering or other signal processing may beperformed on the excitation signal in correspondence with bowacceleration or deceleration parameterization of excitation indicativegestures in the gesture stream.

A composite excitation signal formed as the superposition of variousexcitations mapped from the gesture stream is, in general, commuted withthe modeled impulse response of the instrument as a resonator andintroduced (as an aggregate excitation) as an input to a filtered delayloop wherein coarse and fine pitch control are respectively provided bydelay line length and the loop filter. In synthetic violinimplementations such as described herein, string length indicativegestures are mapped to manipulate pitch controls of the filtered delayloop. Output of the above-described synthesis (754) is converted toacoustic energy by acoustic transducer 512 to produce a syntheticaudible rendering 511 in correspondence with the captured gesture stream551 which expresses the user's performance.

FIG. 6 is a functional block diagram that illustrates capture, encodingand transmission of a gesture stream encoding corresponding to a userperformance on a synthetic violin instrument hosted on a first, portablecomputing device 501 together with receipt of the gesture streamencoding and acoustic rendering of the performance on a second, portablecomputing device 502 (typically remote from device 501) on which aninstance (550A) of the above-described synthetic violin application islikewise hosted. Capture and encoding of a gesture stream (includingstring length indicative gestures and excitation indicative gestures) atportable computing device 501 is as described above with reference toFIG. 5. Likewise, processing of the gesture stream communicated (522)via network(s) 104 to the second device 501 as well as synthesis 554Athereon and rendering (511A) at acoustic transducer 512A thereof arealso as described above with reference FIG. 5 and like facilities(synthesis 554 and acoustic transducer 512) illustrated therein.

It is notable that, in the above-described commuted synthesisimplementation of synthesis 554, it is the combination of string lengthand excitation table repetition rate that makes for the pitch.Specifically, a body impulse response table is fired at the intendedpitch, and the modeled string is set (and successively updated) to anappropriate length in correspondence with parameterization of stringlength indicative gestures from the gesture stream.

FIG. 7 illustrates a variation on the synthetic violin of FIG. 5, inwhich certain string length indicative gestures and excitationindicative gestures are adapted for a synthetic guitar implementationand to more closely correspond to physical instrument analogs familiarto a guitar player. Thus, a six string instrument is illustrated inwhich it will be understood that string length indicative gesturesexpressed by a user are selective for string lengths (of respectivestrings) quantized in accord with frets typical of guitar-typeinstrument. In particular, visual cues for finger contacts suggestive ofchords to be sounded are generated by visual cue generator 757 from amusical score retrieved from storage 556. Although the set of visualcues supplied includes cues suggestive of fingerings for guitar chords,it will e understood that individual string fingerings (and visual cuestherefor) may also e provided. In general, musical score based visualcuing techniques employed in a synthetic guitar implementation will beunderstood with reference to songbook and lesson plan modes previouslydescribed herein for synthetic violin implementations.

As before, while a user may (in general) express and sound whatevernotes he/she desires by way of string length indicative and excitationindicative gestures on multi-touch sensitive display 514, often the userexpresses (or at least attempts to express) a sequence of gestures (here718) generally in correspondence with the visual cues and the musicalscore from which such visual cues are generated (757). Also as beforeand depending on the implementation (or mode of operation), a variety ofinputs (752) may feed the capture/encoding (753) to produce a gesturestream 751 supplied to synthesis 754. Relative to FIG. 7, string lengthindicative gestures (e.g., finger contacts, lateral travel of a fingercontact along a string, etc.) and excitation indicative gestures (e.g.,strum-type finger travel gestures across all or a subset of the stringsas well as pluck or pick-type finger contacts) captured from multi-touchsensitive display 514 are collectively summarized as gestures 718.Additional sources of user gestures may be supported in someembodiments. For example, in some embodiments, an accelerometer 517input may be used to capture gestures (519) indicative of vibrato,trilling or some other ornamentation of the user's performance.

As will be appreciated, string length indicative gestures for fingeringsof respective strings (including implicitly string lengths of openunfingered strings) are illustrated (in gesture sequence 718) forsuccessive E major and D major chords. As will be further appreciated,string excitation indicative gestures (e.g., strum-type finger travel)across all or the subset of strings corresponding to a given chord maybe expressed by the user. In each case, the string length indicative andstring excitation indicative gestures are captured and encoded (753) aresupplied as an input to digital synthesis 754 of the guitar and itsconstituent strings.

Turning to digital synthesis 754, a variety of computational techniquesmay be employed and will be appreciated by persons of ordinary skill inthe art. For example, as before, physical modeling of string vibrationand acoustics of the modeled instrument may be employed in a manneranalogous to that described above with reference to digital synthesis ofa violin. Nonetheless, given the generally simplified excitation andstring length variation opportunities afforded in a guitar-typeinstrument, a commuted synthesis-type physical model of the guitar maybe unnecessary (and indeed computationally excessive) in someembodiments. Instead, computational techniques such as wavetablesynthesis may be employed. Wavetable synthesis techniques are well knownin the art and, any of a variety of suitable techniques may be employed.As with the previously described implementations, string excitationindicative gestures and string length indicative gestures actuallyexpressed by the user drive a wavetable digital synthesis.

FIG. 8 is a screen shot depicting certain aspects of a user interfacedesign for a storybook teaching mode of a synthetic violin instrument inaccordance with some embodiments of the present invention. In additionto basic textual, graphical and feedback assisted instruction regardingfingering techniques, bow contact gestures, etc., storybook lesson planspresent exercises (typically as visually cued performances correspondingto an underlying musical score) at successive levels of difficulty.Grading, or points awarded in a game or competition framework, are basedon correspondence of the user/musician's actual expression (by way ofcaptured string length indicative and string excitation indicativegestures) with note selections, durations, timings and even score codedperformance figures such as vibrato, trills, string bending, etc. Inthis way, correspondence with (or variances from) a desired exercise,lesson, or selection for recital-type competition, including (i) welltimed, premature or late bowing or string contact, (ii) tonal variancefrom the musical score based on actually expressed string contacts,(iii) well or poorly expressed vibrato and/or (iv) desired or undesiredacceleration/deceleration of the bow based on thumb movement may beaccounted for in the form of performance grading and/or feedback.

Exemplary Portable Computing Device Platforms and Network Configurations

FIG. 9 illustrates features of a mobile device that may serve as aplatform for execution of software implementations in accordance withsome embodiments of the present invention. More specifically, FIG. 9 isa block diagram of a mobile device 600 that is generally consistent withcommercially-available versions of iPhone™ and iPod Touch™ mobiledigital devices or, at a larger form factor, with an iPad™ computingtablet, each executing a version of the iOS operating system availablefrom Apple Inc. Although embodiments of the present invention arecertainly not limited to iPhone, iPod or iPad deployments orapplications (or even iOS devices), the iPhone device, together with itsrich complement of sensors, multimedia facilities, applicationprogrammer interfaces and wireless application delivery model, providesa highly capable platform on which to deploy certain implementations.Based on the description herein, persons of ordinary skill in the artwill appreciate a wide range of additional mobile device platforms thatmay be suitable (now or hereafter) for a given implementation ordeployment of the inventive techniques described herein.

Summarizing briefly, mobile device 600 includes a display 602 that canbe sensitive to haptic and/or tactile contact with a user.Touch-sensitive display 602 can support multi-touch features, processingmultiple simultaneous touch points, including processing data related tothe pressure, degree and/or position of each touch point. Suchprocessing facilitates gestures and interactions with multiple fingers,chording, and other interactions. Of course, other touch-sensitivedisplay technologies can also be used, e.g., a display in which contactis made using a stylus or other pointing device.

Typically, mobile device 600 presents a graphical user interface on thetouch-sensitive display 602, providing the user access to various systemobjects and for conveying information. In some implementations, thegraphical user interface can include one or more display objects 604,606. In the example shown, the display objects 604, 606, are graphicrepresentations of system objects. Examples of system objects includedevice functions, applications, windows, files, alerts, events, or otheridentifiable system objects. In some embodiments of the presentinvention, applications, when executed, provide at least some of thedigital acoustic functionality described herein.

Typically, the mobile device 600 supports network connectivityincluding, for example, both mobile radio and wireless internetworkingfunctionality to enable the user to travel with the mobile device 600and its associated network-enabled functions. In some cases, the mobiledevice 600 can interact with other devices in the vicinity (e.g., viaWi-Fi, Bluetooth, etc.). For example, mobile device 600 can beconfigured to interact with peers or a base station for one or moredevices. As such, mobile device 600 may grant or deny network access toother wireless devices.

Mobile device 600 includes a variety of input/output (I/O) devices,sensors and transducers. For example, a speaker 660 and a microphone 662are typically included to facilitate audio, such as the audiblerendering of musical performances as described elsewhere herein. In someembodiments of the present invention, speaker 660 and microphone 662 mayprovide appropriate transducers for techniques described herein. Anexternal speaker port 664 can be included to facilitate hands-free voicefunctionalities, such as speaker phone functions. An audio jack 666 canalso be included for use of headphones and/or a microphone.

Other sensors can also be used or provided. A proximity sensor 668 canbe included to facilitate the detection of user positioning of mobiledevice 600. In some implementations, an ambient light sensor 670 can beutilized to facilitate adjusting brightness of the touch-sensitivedisplay 602. An accelerometer 672 can be utilized to detect movement ofmobile device 600, as indicated by the directional arrow 674.Accordingly, display objects and/or media can be presented according toa detected orientation, e.g., portrait or landscape. In someimplementations, mobile device 600 may include circuitry and sensors forsupporting a location determining capability, such as that provided bythe global positioning system (GPS) or other positioning systems (e.g.,systems using Wi-Fi access points, television signals, cellular grids,Uniform Resource Locators (URLs)) to facilitate geocodings describedherein. Mobile device 600 can also include a camera lens and sensor 680.In some implementations, the camera lens and sensor 680 can be locatedon the back surface of the mobile device 600.

Mobile device 600 can also include one or more wireless communicationsubsystems, such as an 802.11b/g communication device, and/or aBluetooth™ communication device 688. Other communication protocols canalso be supported, including other 802.x communication protocols (e.g.,WiMax, Wi-Fi, 3G), code division multiple access (CDMA), global systemfor mobile communications (GSM), Enhanced Data GSM Environment (EDGE),etc. A port device 690, e.g., a Universal Serial Bus (USB) port, or adocking port, or some other wired port connection, can be included andused to establish a wired connection to other computing devices, such asother communication devices 600, network access devices, a personalcomputer, a printer, or other processing devices capable of receivingand/or transmitting data. Port device 690 may also allow mobile device600 to synchronize with a host device using one or more protocols, suchas, for example, the TCP/IP, HTTP, UDP and any other known protocol.

FIG. 10 illustrates instances (501, 502) of a portable computing devicesuch as mobile device 600 programmed with user interface code, gesturecapture code and digital synthesis code for a synthetic stringinstrument in accord with the functional descriptions herein. Deviceinstance 501 operates in a visual cuing and performance capture mode andsupplies a gesture stream to device instance 502 for local digitalsynthesis and audible rendering. Device instance 501 may also supply oneor more devices e.g., mobile device instance 501, laptop computer 101 orserver 102 with a rendering of the user's performance to a mediaencoding. Illustrated devices communicate (and data described here iscommunicated therebetween) using any suitable wireless data (e.g.,carrier provided mobile services, such as GSM, 3G, CDMA, WCDMA, 4G,4G/LTE, etc. and/or WiFi, WiMax, etc.) including any interveningnetworks 104 using facilities (exemplified as access point 107 and telcotower 108, and server 102).

Other Embodiments

While the invention(s) is (are) described with reference to variousembodiments, it will be understood that these embodiments areillustrative and that the scope of the invention(s) is not limited tothem. Many variations, modifications, additions, and improvements arepossible. For example, while particular violin- and guitar-typeembodiments, user interface constructs and gesture sets have beenemphasized in the description, other multi-string instruments and othervariations on the user interfaces and gesture sets will be appreciatedbased on the description herein. For example, although syntheticmulti-string instrument have been depicted with user interface andgesture constructs that visually present a multiplicity of strings, itwill be understood that some variations may devolve a multiplicity ofparallel strings and associated visual cues onto a single string userinterface and visual cuing model. In such single string variations onthe embodiments more exhaustively described herein, it will beappreciated that notes that (in the physical analog) would be sounded onrespective ones of the plural strings are instead coded and expressed byfinger contacts (typically as a laterally displaced tonal scale) on asingle string visual representation.

Likewise, while certain illustrative signal processing techniques havebeen described in the context of certain illustrative applications,persons of ordinary skill in the art will recognize that it isstraightforward to modify the described techniques to accommodate othersuitable signal processing techniques and effects. Particular commutedsynthesis physical models and wavetable synthesis techniques referencedherein are merely exemplary.

Embodiments in accordance with the present invention may take the formof, and/or be provided as, a computer program product encoded in amachine-readable medium as instruction sequences and other functionalconstructs of software, which may in turn be executed in a computationalsystem (such as a iPad, iPhone, iPod Touch handheld, mobile device orportable computing device) to perform methods described herein. Ingeneral, a machine readable medium can include tangible articles thatencode information in a form (e.g., as applications, source or objectcode, functionally descriptive information, etc.) readable by a machine(e.g., a computer, computational facilities of a mobile device orportable computing device, etc.) as well as tangible storage incident totransmission of the information. A machine-readable medium may include,but is not limited to, magnetic storage medium (e.g., disks and/or tapestorage); optical storage medium (e.g., CD-ROM, DVD, etc.);magneto-optical storage medium; read only memory (ROM); random accessmemory (RAM); erasable programmable memory (e.g., EPROM and EEPROM);flash memory; or other types of medium suitable for storing electronicinstructions, operation sequences, functionally descriptive informationencodings, etc.

In general, plural instances may be provided for components, operationsor structures described herein as a single instance. Boundaries betweenvarious components, operations and data stores are somewhat arbitrary,and particular operations are illustrated in the context of specificillustrative configurations. Other allocations of functionality areenvisioned and may fall within the scope of the invention(s). Ingeneral, structures and functionality presented as separate componentsin the exemplary configurations may be implemented as a combinedstructure or component. Similarly, structures and functionalitypresented as a single component may be implemented as separatecomponents. These and other variations, modifications, additions, andimprovements may fall within the scope of the invention(s).

1. A method comprising: using a first portable computing device as asynthetic string instrument; presenting on a multi-touch sensitivedisplay of the portable computing device, and in correspondence with amusical score, temporally synchronized visual cues relative torespective strings of the synthetic string instrument; capturing usergestures indicative of length of respective strings of the syntheticstring instrument from data sampled in correspondence with respectivefinger contacts with the multi-touch sensitive display along visualdepictions of the respective strings; capturing user gestures indicativeof excitation of at least one of the strings; encoding a gesture streamfor a performance of the user by parameterizing at least a subset of thestring length and string excitation indicative user gestures; andaudibly rendering the performance on the portable computing device usingthe encoded gesture stream as an input to a digital synthesis of thesynthetic string instrument executing on the first portable computingdevice, wherein the captured gesture stream, and not the musical scoreitself, drives the digital synthesis.
 2. The method of claim 1, whereinthe string excitation indicative gestures are distinct from the stringlength indicative gestures.
 3. The method of claim 1, wherein thedigital synthesis includes a model of acoustic response for one of aviolin, a viola, a cello and a double bass; and wherein the capturedstring excitation indicative gestures include a bow traversal gestureindicated by contact of the user's thumb or finger with a rotatingvisual on the multi-touch sensitive display.
 4. The method of claim 3,wherein radial excursion of the user's thumb or finger contact with therotating visual is indicative of bow speed.
 5. The method of claim 3,wherein the captured string excitation indicative gestures furtherinclude pluck- or strum-type gestures indicated by contact of the user'sthumb or finger with respective visual indications of the strings. 6.The method of claim 1, wherein the digital synthesis is of one of aguitar, banjo, ukulele, lute, mandolin or sitar; and wherein thecaptured string excitation indicative gestures include pluck- orstrum-type gestures indicated by contact of the user's thumb or fingerwith respective visual indications of the strings.
 7. The method ofclaim 6, wherein the digital synthesis includes wavetable or frequencymodulation synthesis in correspondence with the parameterization of atleast the string length indicative user gestures.
 8. The method of claim6, wherein the digital synthesis includes physical model of acousticresponse for the guitar, banjo, ukulele, lute, mandolin or sitar.
 9. Themethod of claim 1, further comprising: capturing a damping or mutingindicative gesture expressed by the user; and in the digital synthesis,attenuating in correspondence with a parameterization of the damping ormuting indicative gesture.
 10. The method of claim 1, furthercomprising: responsive to a user selection on the multi-touch sensitivedisplay, retrieving an encoding of the musical score via thecommunications interface.
 11. The method of claim 1, further comprising:grading the user's performance based at least in part on correspondenceof captured finger contact gestures with notes or positions and timingsencoded in the musical score.
 12. The method of claim 1, furthercomprising: grading the user's performance based at least in part oncorrespondence of captured finger contact gestures with vibrato,trilling or other spatio-temporal effects encoded in the musical score.13. The method of claim 1, further comprising: presenting on themulti-touch sensitive display a lesson plan of exercises, wherein thecaptured gesture stream corresponds to performance by the user of aparticular one of the exercises; and advancing the user to a nextexercise of the lesson plan based on a grading of the user's performanceof the particular exercise.
 14. A method comprising: using a firstportable computing device as a synthetic string instrument; capturinguser gestures relative to respective strings of the synthetic stringinstrument from data sampled in correspondence with respective fingercontacts with a multi-touch sensitive display of the portable computingdevice; capturing user gestures indicative of bow traversal of at leastone of the strings; encoding a gesture stream for a performance of theuser by parameterizing at least a subset of events captured from thefinger contacts and bow traversal; and audibly rendering the performanceon the portable computing device using the encoded gesture stream as aninput to a digital synthesis of the synthetic string instrumentexecuting on the first portable computing device.
 15. The method ofclaim 14, further comprising: presenting on the multi-touch sensitivedisplay, and in correspondence with a musical score, temporallysynchronized visual cues to guide the user's gestures relative to therespective strings of the synthetic string instrument, wherein thegesture stream captured from the user's performance, and not the musicalscore itself, drives the digital synthesis.
 16. The method of claim 15,wherein the user gestures indicative of bow traversal are captured incorrespondence with thumb or finger contact with the multi-touchsensitive display.
 17. The method of claim 15, further comprising:dynamically correcting captured finger contact gestures in accord withnotes or positions encoded in the musical score.
 18. The method of claim15, further comprising: grading the user's performance based at least inpart on correspondence of captured finger contact gestures with notes orpositions and timings encoded in the musical score.
 19. The method ofclaim 15, further comprising: grading the user's performance based atleast in part on correspondence of captured finger contact gestures withvibrato, trilling or other spatio-temporal effects encoded in themusical score.
 20. The method of claim 15, further comprising:manipulating a second portable device relative to the first, wherein theuser gestures indicative of bow traversal include lateral movement andinclination of the second portable device captured in correspondencewith the manipulation.
 21. The method of claim 20, wherein the secondportable computing device includes either or both of multi-axisaccelerometer and a gyroscopic sensor for capture of orientation andmotion dynamics of the second portable computing device, and wherein thesecond portable computing device is configured to compute and wirelesslycommunicate to the first portable computing device the bow traversalgestures based on the captured orientation and motion dynamics.
 22. Themethod of claim 20, further comprising: capturing at a multi-touchsensitive display of the second portable computing device further usergestures indicative of one or more of bow pressure and bow-on-string.23. The method of claim 20, further comprising: grading the user'sperformance based at least in part on correspondence of one or moreparameterizations of the captured bow traversal gestures with bowingtechniques encoded in the musical score, wherein the one or moreparameterizations are selected from the set of speed of travel, positionof a sounding point, and method of attack.
 24. The method of claim 15,further comprising responsive to a user selection on the multi-touchsensitive display, retrieving an encoding of the musical score via thecommunications interface.
 25. The method of claim 15, furthercomprising: presenting on the multi-touch sensitive display a lessonplan of exercises, wherein the captured gesture stream corresponds toperformance by the user of a particular one of the exercises; andadvancing the user to a next exercise of the lesson plan based on agrading of the user's performance of the particular exercise.
 26. Themethod of claim 14, wherein the first portable computing device includesa communications interface, the method further comprising, transmittingthe encoded gesture stream via the communications interface forrendering of the performance on a remote device.
 27. The method of claim26, wherein the remote device includes a server from which the renderedperformance is subsequently supplied as one or more audio encodingsthereof.
 28. The method of claim 26, further comprising: audiblyrendering a second performance on the portable computing device using asecond gesture stream encoding received via the communications interfacedirectly or indirectly from a second remote device, the secondperformance rendering using the received second gesture stream encodingas an input to the digital synthesis of the musical instrument.
 29. Themethod of claim 26, further comprising: geocoding the transmittedgesture stream; and displaying a geographic origin for, and incorrespondence with audible rendering of, a third performance encoded asa third gesture stream received via the communications interfacedirectly or indirectly from a third remote device.
 30. The method ofclaim 14, wherein the digital synthesis includes a model of acousticresponse for one of a violin, a viola, a cello and a double bass. 31.The method of claim 20, wherein the first and second portable computingdevices are each selected from the group of: a compute pad, a laptopcomputer, notebook computer or netbook; a personal digital assistant orbook reader; and a mobile phone or media player.
 32. A computer programproduct encoded in one or more media, the computer program productincluding instructions executable on a processor of the portablecomputing device to cause the portable computing device to perform themethod of claim
 14. 33. The computer program product of claim 32,wherein the one or more media are readable by the portable computingdevice or readable incident to a computer program product conveyingtransmission to the portable computing device.
 34. An apparatuscomprising: a portable computing device having a multi-touch displayinterface; and machine readable code executable on the portablecomputing device to implement the synthetic musical instrument, themachine readable code including instructions executable to capture both:(i) user gestures relative to respective strings of the synthetic stringinstrument from data sampled in correspondence with respective fingercontacts with a multi-touch sensitive display of the portable computingdevice and (ii) user gestures indicative of bow traversal of at leastone of the strings and to encoded a gesture stream for a performance ofthe user by parameterizing at least a subset of events captured from thefinger contacts and bow traversal; and the machine readable code furtherexecutable to audibly render the performance on the portable computingdevice using the encoded gesture stream as an input to a digitalsynthesis of the synthetic string instrument executing on the portablecomputing device.
 35. The apparatus of claim 34, configured towirelessly communicate with a second portable computing device proximatethereto, the second portable computing device including either or bothof multi-axis accelerometer and a gyroscopic sensor for capture oforientation and motion dynamics of the second portable computing device,and machine readable code executable on the second portable computingdevice to compute and wirelessly communicate to the first portablecomputing device the bow traversal gestures based on the capturedorientation and motion dynamics.
 36. The apparatus of claim 34, embodiedas one or more of a compute pad, a handheld mobile device, a mobilephone, a laptop or notebook computer, a personal digital assistant, asmart phone, a media player, a netbook, and a book reader.
 37. Acomputer program product encoded in media and including instructionsexecutable to implement a synthetic musical instrument on a portablecomputing device having a multi-touch display interface, the computerprogram product encoding and comprising: instructions executable tocapture both: (i) user gestures relative to respective strings of thesynthetic string instrument from data sampled in correspondence withrespective finger contacts with a multi-touch sensitive display of theportable computing device and (ii) user gestures indicative of bowtraversal of at least one of the strings and to encode a gesture streamfor a performance of the user by parameterizing at least a subset ofevents captured from the finger contacts and bow traversal, furtherinstructions executable to audibly render the performance on theportable computing device using the encoded gesture stream as an inputto a digital synthesis of the synthetic string instrument executing onthe portable computing device.